Integrating information theory and adversarial learning for cross-modal retrieval

Accurately matching visual and textual data in cross-modal retrieval has been widely studied the multimedia community. To address these challenges posited by heterogeneity gap semantic gap, we propose integrating Shannon information theory adversarial learning. In terms of integrate modality classification entropy maximization adversarially. For this purpose, a classifier (as discriminator) is built to distinguish text image modalities according their different statistical properties. This discriminator uses its output probabilities compute entropy, which measures uncertainty it performs. Moreover, feature encoders generator) project uni-modal features into commonly shared space attempt fool maximizing entropy. Thus, gradually reduces distribution discrepancy features, thereby achieving domain confusion state where cannot classify two confidently. reduce Kullback-Leibler (KL) divergence bi-directional triplet loss are used associate intra- inter-modality similarity between space. Furthermore, regularization term based on KL-divergence with temperature scaling calibrate biased label caused imbalance issue. Extensive experiments four deep models benchmarks conducted demonstrate effectiveness proposed approach.